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VlGiLANCE MONITORING SYSTEM 

(Q The present Invention relates to a vigilance monitoring system. In 
particular the invention relates to a system for monitoring, recording and/or 
analysing vigilance, alertness or wakefulness and/or a stressed state of an 
operator of equipment or machinery in a variety of situations including situations 
wherein the degree of vigilance of the operator has implications for the safety or 
well being of the operator or other persons. A typical application may Include 
monitoring the driver of a vehicle or pilot of an aircraft, although the Invention also 
has applications In areas Involving related occupations such as train drivers and 
operators of equipment such as cranes and industrial machinery in general, and 
where lack of operator vigilance can give rise to hannful social or economic 
consequences. 

The system of the present invention will be described herein with reference 
to monitoring a driver of a vehicle nevertheless it is not thereby limited to such 
applications. For example, other applications may include monitoring routine, 
acute or sub-acute physiological parameters of a person or subject In a home, 
work, clinic or hospital environment. The monitored parameters may include 
cardiac, respiratory and movement parameters as well as parameters relating to 
apnea events, subject sleep states or sudden death syndrome on-set. 

The monitoring system is designed. Inter alia, to provide non-Invasive 
monitoring of a driver's physiological data including movement activity, heart 
activity, respiration and other physiological functions. The monitored 
physiological data may undergo specific analysis processing to assist in 
determining of the driver's state of vigilance. The system is designed to detect 
various states of the driver's activity and detect certain conditions of driver fatigue 
or relaxation state that could lead to an unsafe driving condition or conditions. 

system of the present Invention Includes means for gathering 
movement data associated with the driver. The movement gathering means may 
include a plurality of sensors such as touch sensitive mats placed In locations of 

^ ' ■ ■ ■ > 

the vehicle t hat make contact with the driverr su ch as the seat, steering wheel, 
pedal(s), seat belt or the like. Each location may include several sensors or mats 
to more accurately monitor movements of the driver. 
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Signals from the various sensors/mats may be processed and analysed by 
^ a processing means. The processing means may Include a digital computer. 
The processing means may be programmed to recognize particular movement 
signatures or pattems of movement, driver posture or profile and to interpret 
5 these to indicate that vigilance has deteriorated or is below an acceptable 
threshold. The processing means may include one or more algorithms. 

The sensors or mats may include piezoelectric, electrostatic, piezo ceramic 
or strain gauge material. The latter may be manufactured by separating two 
conductive materials such as aluminium foil with an electrolyte material which is 
capable of passing AC but not DC current. In one form the sensors or mats may 
include Capacitive Static Discharge (CSD) or Polyvinylidene fluoride (PVDF) 
material. The sensors/mats may be covered with a non-obtrusive, flexible surface 
which is capable of detecting pressure and/or monitoring electrophysiological 
activity. 

The pressure detecting capability may be used for detecting driver 
movement. The or each sensor may produce an output signal that represents the 
magnitude of the pressure or force that is applied to the sensor. The or each 
pressure signal may thus represent an absolute or quantitative measure of 
pressure applied to the sensor. The electrophysiological activity may include 
electrical signals generated by the body of the driver eg. electrical muscle activity 
and/or pulse activity. 

The sensors or mats may be located in various parts of a vehicle. The 
seat of the driver may be divided Into several sections such as upper or back and 
lower or seat. The upper or back sectiojupay jnclu de sensor^ jnJheJoDedge, 
centreand.base . The lower or seat_ section may includT^nsors in thTfront 
edge, centre and rear. The or each sensor may include CSD or PVDF material. 

The steering wheel may include a plurality of sensors. The steering wheel 
may be divided into eight zones such as upper, upper left, upper right, left, right, 
lower left, lower right and lower. At least one sensor may be associated with 
each zone. The or each sensor may include CSD or PVDF material. 

The floor covering such as carpet may Include a plurality of sensors. The 
floor covering or carpet may be divided into a plurality of zones. At least one 
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sensor may be associated with each zone. The or each sensor may include CSD 
or PVDF material. 

f I The accelerator, clutch and brake pedals may include a plurality of 

sensors. Each pedal may be divided into a plurality of zones such as upper. 
5 middle and lower. At least one sensor may be associated with each zone. The 
or each sensor may include CSD, PVDF or other movement sensitive material. 

The seat belt may Include one or a plurality of sensors. In one form a 
sensor or sensors may be embedded In the fixed (i.e. non-retractable) section of 
the seat belt. The or each sensor may include CSD or PVDF material. 
10 In some embodiments a head tilt device incorporating a positional switch or 

the like may be associated with the drivers cap» glasses or goggles or may be 
arranged to clip over the drivers ear or glasses. The head tilt device may be 
adapted to provide a signal or data which alters in accordance with the position of 
the driver's head. Alternatively a radio tracking device may determine and track a 
1 5 subject's head movements. 

In critical applications of vigilance monitoring including applications 
involving pilots of aircraft, persons responsible for navigating/controlling shipping 
and drivers of road or rail transport it may be desirable to utilize more 
comprehensive methods of vigilance monitoring. The latter may include 
20 techniques used in conventional steep monitoring. A head band and/or chin band 
sensor may be used to monitor EE6, EMG and EGG signals. The head band 
sensor may include separate left and right frontal zones and left and right eye 
zones. The sensor may Include CSD or PVDF material or other material sensitive 
to measuring patient skin electrical surface variations and/or impedance. 
25 Various sensors/techniqiies may be adapted for monitoring eye movement 

including those based on reflected light, electric skin potential, contact lenses, 
limbus tracking, video imaging and magnetic induction. The sensors/techniques 
may include EOG electrodes, infrared detection of eye movements and/or video 
tracking and processing of eye movements. The sensors/techniques may be 
30 adapted for monitoring the left eye only or the right eye only or both eyes. 

Raw data which is collected from the various sensors positioned around 
the vehicle may be filtered and amplified prior to processing and analysis. A 
significant purpose of the processing and analysis Is to determine the driver's 
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State of vigilance, alertness or wakefulness. In some embodiments, the system 
may be adapted to effect remedial action, le. the system may take steps to alert 
the driver or to actively Intervene In the control of the vehicle, when it Is deemed 
that such action is warranted or desirable. 

^ ^ Processing of data may be perfomied In several stages. Including primary, 
secondary and tertiary analysis. 

Primary analysis refers to processing of raw data from the various sensors. 
This raw data may be filtered and amplified prior to analog to digital conversion. 
Primary analysis may be adapted to determine valid body movements of the 
driver as distinct from spurious signals and artefacts due to environmental factors 
including noise. 

\(!^ Valid body movements may be determined by applying a combination of 

processing techniques Including: 

1. signal threshold detection whereby signals below or above a pre- 
determined threshold are Ignored and/or classified as noise or 
artefact. 

VI 2. frequency filtering whereby high-pass, low-pass and notch filters are 

adapted to remove noise and artefact signals, 
\ I 3. signal compression whereby data Is minimized by presenting main 

data points such as signal peaks, troughs, averages and zero 

crossings, 

■l^^4. half period, amplitude analysis of signals, including analysis as 
disclosed in AU Patent 632932 entitled "Analysis System for 
Physiological Variables", assigned to the present applicant, the 
disclosure of which is Incorporated herein by cross reference. 
. -\Jt^Threshold detection may facilitate distinguishing random and non- 
significant electrical noise (typically spikes of small duration) relative to signals 
representing valid or actual body movements. Threshold detection may apply to 
both amplitude and duration of the signals. The relevant threshold(s) may be 
determined from clinical trials and/or historical data. Where the detection is 
based on amplitude it may be determined in both negative and positive phases of 
the signal. Amplitude detection may be based on a measurement of the peak-to- 
peak signal. Alternatively, the positive and negative peak amplitudes can be 
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measured separately. Threshold detection may be combined with a form of zero* 
base line detection so that electronic offsets do not adversely affect the accuracy 
of threshold detections. Each body movement which exceeds the predetermined 
amplitude and/or duration may be classified as an event for further processing. 

Secondary analysis may be adapted to process the results of primary 
analysis. Secondary analysis may process data for the purpose of presentation 
and/or display. Data may be displayed or printed in a tabular, graphical or other 
format which facilitates interpretation of the data. One purpose of the 
representation and/or display is to represent a driver's state of vigilance and/or 
fatigue. In one form each event identified during primary analysis may be 
counted for a fixed period of time or epoch. The fixed period of time may be 30 
seconds or 60 seconds, or other period which 16 adequate for determining body 
movement trends. The count value or number of valid body movements in a 
given period (eg. 30 seconds) may be represented in a display as, say, the length 
of a vertical bar. 



'2^1^ Where it is desired to display the energy or power associated with valid 
body movements in a particular epoch or time period, the average amplitude 
associated with each event may be indicated by the length of the vertical bar 
whilst the count value or number of valid body movements for each epoch may be 
represented by colouring the vertical bar. For example the colours green, blue, 
yellow, orange, red may indicate count values or movement numbers in 
ascending order ie. green indicating the lowest movement number for a particular 
epoch and red indicating the highest movement number for a particular epoch. 
Altematively, data may be displayed on a 3 dimensional graph wherein for 
example the x dimension of the graph represents time or epochs, the y dimension 
represents the average amplitude, while the z dimension represents the number 
of events during a particular epoch. The above display techniques, may facilitate 
interpretation of the number of valid body movements and the amplitude of those 
movements and association of this data with the driver's activity or state of 
vigilance, alertness or wakefulness. 

It may also be relevant to measure the period of each individual body 
movement as this may provide an indication of the energy that is associated with 
the movement. For example, if a driver squeezes the steering wheel in a rapid 
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response as distinct from gripping the wheel as part of a focussed steering 
manoeuvre, the pattern of signals in each case will be different. The rapid 
response may appear as a small cluster of movements/signals or as a single 
movement/signal with a relatively short duration or period of time. In contrast, the 
steering manoeuvre may appear as a larger cluster of movements/signals over a 
relatively longer period of time or as a single movement/signal having a relatively 
long duration. 

\^ The type of signal which may be expected (cluster or single 
movement/signal) will depend in part upon the type of sensor. For example, 
piezo ceramic or PVDF sensors may emit fewer clusters of signals but may emit 
signals with larger time periods in relation to the actual period of the movement 
which is being monitored. A capacltive electrostatic sensor is more likely to emit 
clusters of "spikes" being relatively short period signals. It may be necessary to 
record the energy level of each movement as this energy level may fall below a 
certain threshold when the driver is in a fatigued state. If. for example the driver 
has relaxed, then the energy of a body movement in the actions of driving may be 
significantly more subdued than in the case where the driver is alert, and his 
muscle activity is significantly greater. Therefore it may be useful to measure and 
record each and every body movement. This data could be displayed on high- 
resolution graphs where for example the X-axis represents Vz second periods and 
960 lines make up each continuous section- or 480 seconds (8 minutes). The 
largest amplitude signal in each second period could then be displayed on the 
X-Axis. The Y-Axis on the other hand could be represented by a scale of 
amplitudes representing each body movement. This graph would be more 
precise in representing the actual signal level of each body-movement and the 
subsequent muscle status for a driver. 

It may also be useful to detect events that are represented by groups of 
movements, where, for example, the groups of movements may be indicative of a 
driver activity of interest. Detection of groups of movements may include user 
configurable or preset values for; 

the maximum time between consecutive body-movements in order to 
qualify as being counted as part of a periodic body-movement. 
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♦ the number of consecutive body-movements that are required to qualify for 
a periodic movement. 

♦ the time period during which this number of body-movements must exist in 
^ order to qualify as a periodic body-movement 

5 ^ -0 Periodic measurement analysis can detect, for example, absence of 
movements which can be associated with a driver's fatigue. 

^1 1^^^ Tertiary analysis may be adapted to process the results of secondary 
analysis. One purpose of tertiary analysis is to determine the status of a drivers 
state of vigilance, alertness or wal<efulness. Tertiary analysis may process the 

10 results of secondary analysis to produce intermediate data and/or indicate trends 
in the data. The intermediate data and trends may be used to provide summary 
reports and further tabular and/or graphic representations of a drivers status or 
condition. The intermediate data may be processed by one or more vigilance 
algorithms to determine the status of a driver's vigilance, alertness or 

15 wakefulness. Intermediate data of various types may be derived and the 
vigilance algorithm(s) may make use of such data to deterrriine the status of the 
driver. The intermediate data may include: 

♦ Rate of change of body movement detections 

♦ Rate of change of body movement amplitudes 

20 ♦ Area under curve of time versus body movement, for various sequential 
epochs to detect trends of subject movement changes (amplitude or 
number of movements) 

♦ Correlation of sensor data for patterns of amplitude, energy and body 
movement changes that can be associated with driver fatigue 

25 ♦ Change in frequency of body movement signals 

♦ Change in amplitude periods of body movement signals 

♦ Change in phase relationships of body movement signals 

♦ Relative phase relationship between each section and other types of 
sensor sections. 

30 /^^^ Following tertiary analysis the vigilance algorithm(8) may be adapted to 
correlate the intermediate data and/or apply combinational logic to the data to 
detect patterns of movement (or lack thereof) which, based on historical data or 
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clinical trials. Indicates that the driver is or may be excessively relaxed or Is below 
an acceptable threshold of vigilance, alertness or wakefulness. 
^\^The vigilance algorithm(s) may Incorporate one or more look up tables 
including reference movement data and default values associated with 
5 acceptable and unacceptable levels of driver fatigue. Histograms including 
movement histograms of the kind described in AU Patent 632932 based on the 
work of Rechitschaffen and Kayles (R & K) may be used as well as tables 
showing weighted values and actual movement data for each sensor. 

The vigilance algorithm(s) may determine a vigilance probability factor 
10 (0-100%) as a function ofweighted movement data values. 

Upon detecting a vigilance probability factor which is below an acceptable 
threshold, the system may be arranged to Intervene In the control of the vehicle or 
to alert the driver of the vehicle and/or other vehicles. Vehicle control Intervention 
may include restriction of speed, controlled application of brakes, cutting-off fuel 
and/or disabling the accelerator pedal. Driver alerting intervention may include 
use of sprays designed to stimulate the driver, vibrating the steering wheel, seat 
belt or floor area in the vicinity of the driver, an audible alarm and/or use of bright 
cabin lights. The driver can also be alerted by winding down the driver window 
and/or other effective alerting methods as may be applicable to each individual 
driver. Drivers of other vehicles may also be alerted by means of flashing hazard 
lights and/or sounding of a siren. Vehicle control intervention may be integrated 
with and form part of a vehicle control system or it may be Interfaced to an 
existing vehicle control system. Vehicle control intervention may be interfaced 
with GSM or other communication systems to provide eariy warning indication 
that a driver or operator of equipment is in a stressed, fatigued or other 
undesirable condition that may be detected. 

To assist differentiating normal and acceptable driver vigilance from 
fatigued or inappropriate driver conditions, calibration of the various sensor and 
transducer outputs is possible. Calibration can set the system's detection 
parameters In accordance with varying driver movement and other driver signals. 
Calibration is beneficial because driver sensor and signal outputs will vary with 
different drivers. " Background noise will also vary with different vehicles. The 



20 



wo 00/44580 PCT/AU99/01166 

-9- 

need for calibration may be proportional to the critical nature of the driving or 
dependent on the level of accuracy required for fatigue monitoring and detection. 

The need for calibration may to some extent be removed by utilizing 
artificial intelligence to distinguish baseline conditions for a drivers normal wakeful 
5 state to facilitate subsequent analysis and determining when a driver's state 
indicates tatigue or lapse of vigilance. Artificial Intelligence may be emtxxiled in 
one or more automated systems including one or more mathematical algorithms. 
Artificial intelligence includes the systems ability to self-learn or teach itself 
conditions associated with the driver which constitute normal or alert driving as 

10 distinct from conditions which constitute abnomial or fatigued driving. 

Artificial intelligence may allow the driver of a specific vehicle to select a 
mode of operation during which the driver's movements during normal or wakeful 
driving are monitored and diagnosed in order to determine typical thresholds and 
correlations between various sensors, for the purpose of determining true fatigue 

15 states of the driver as distinct from alert states of the driver. Artifteial intelligence 
may also facilitate adaptation of the vigilance algorithm(s), to the specific vehicle's 
background noise characteristics. 

Artificial intelligence may include different response patterns for correlating 
movement data from the various sensors for distinguishing valid driver 

20 movements from environmental vibrations and noise. These may be classified 
and described by. for example, a look up table that records expected patterns or 
combinations of signals for different cases of environmental noise as distinct from 
driver generated signals. For example. If the driver moves his hand, signals from 
sensors in the steering wheel and arm sections of the seat may correlate 

25 according to a specific pattern. Alternatively, if the vehicle undergoes a severe or 
even subtle vibration due to road or engine effects, a broader range of sensors 
may be similariy affected and this may be manifested as amplitudes which follow 
predetermined correlation patterns. Signals from the sensors may increase in 
strength or amplitude according to the proximity of the source of the sound or 

30 vibrations. Where the source of the vibration Is localized, this may manifest itself 
as a pattern of similar waveforms across the various sensors which reduce 
progressively in amplitude as the sensor's distance from the source increases. 
For example, if the source of the vibration is road noise, the floor sensors may 
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reglster maximum amplitude whereas the steering wheel sensors which are 
furthest from the road noise may register minimum amplitude. 
1^ The phase relationship of vibrations from various sources may also provide 
some guide as to the likely source of the vibrations. For example, if the vibrations 
emanate from the driver's movement then it is more likely that several signals with 
similar phase may be detected. On the other hand, if the signals have varying 
phase relationships, then It is more likely that the source of the vibrations giving 
rise to these signals is random as may be expected if the vibrations emanate from 
the vehicle environment. 

^ Similar phase signals arising from driver movements may be distinguished 
from similar phase signals arising from artefacts or the vehicle environment by 
relating the environmental noise to sensors located near sources of expected 
noise In the vehicles, eg. engine noise, wheel noise, and other vibrations and 
noise. This may be detected by carefully locating microphones and vibration 
15 sensors in the vehicle. 

r^> Cancellation of environmental noise can be assisted by monitoring signals 
from the microphones and sensors with a view to applying the most effective 
signal cancellation techniques In order to reduce as much as possible the artefact 
or noise effects or unwanted signals within the vehicle environment. 

^One example of the application of noise cancellation techniques includes 
detection of the various road bumps and Ignoring the effect of these bumps on 
the data being analysed from the various vehicle sensors of interest. 

Another example of the application of noise cancellation techniques 
includes detection of various engine noises and application of a signal of opposite 
phase to the motor noise in order to cancel the artefact. One example of phase 
cancellation techniques which may be adopted is disclosed in PGT application 
AU97/00278. the disclosure of which Is incorporated herein by cross-reference. 

^.^Other examples of noise cancellation include filtering wherein highpass. 
lowpass and notch filters may be used to assist artefact removal. 

Artificial Intelligence may learn to Ignore periodic signals from sensors in 
the vehicle as these are likely to arise from mechanical rotations within the 
vehicle, thus improving the separation of artefact signals from signals of interest, 
such as signals which indicate true driver movement. 
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Artificial intelligence may also learn to recognize changes in the driver's 
state which reflect changes in driver vigilance or wakefulness. Points of 
calculation and analysis of sensor data for the purpose of comparison and 
correlation with previously monitored data may include: 
5 spectral arialysls of signals with a range of consecutive time periods; 

Yz period time amplitude analysis of signals and other techniques used in 

conventional sleep analysis as disclosed in AU Patent 632932; 

calculation of the number of movements per consecutive periods of time, 

wherein the consecutive periods of time may typically be, 1 second or Yz 
10 second; 

calculation of average signal levels during periods of, say. 20 or 30 

seconds; 

calculation of total "area under the curve" or integration of sensor signals 
for a period of, say, 20 or 30 seconds; 
15 correlation and relationship between various combinations of input sensor 

channels; 

ECG heart rate and respiration signals, the latter signals providing an 
indication of the driver's wakeful state, as heart-rate and respiration signals 
during the sleep state are well documented in a number of medical 
20 joumals. 

Artificial intelligence may be applied in conjunction with pressure sensors 
in vehicle seats and/or seat belts to control air bag deployment. In this way air 
bag deployment may be restricted for children or validated with different crash 
conditions for children and adults. For example, if a child is not detected as t>eing 
25 thrust fonvard by means of pressure data received from seat/seat belt sensors, 
deployment of air bags and possible air bag injury to the child may be avoided. 

Deployment of air bags may generally be validated more intelligently by 
analysing data relating to passenger or driver posture, movement, thrust, body 
movement, unique driver or passenger *yav/ etc. 
30 The system may include means for testing a driver's response times. Such 

tests may, if carried out at regular intervals, pre-empt serious driving conditions as 
can be brought about by driver fatigue or a lapse in vigilance. The testing means 
may be adapted to provide a simple method for prompting the driver and for 
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testing the driver's response time. The response test may. for example, request 
the driver to respond to a series of prompts. These prompts may include 
requesting the driver to squeeze left or right hand sections of the steering wheel 
or squeeze with both left and right hands at the same time In response to a 
prompt. The means for prompting the driver may Include, for example, LEDs 
located on the dash of the vehicle or other position that the driver is visually 
aware of. A left LED blinking may for example, prompt the driver to squeeze the 
left hand on the steering wheel. A right LED blinking may prompt the driver to 
squeeze the right hand on the steering wheel. The centre LED blinking may 
prompt the driver to squeeze both hands on the steering wheel. Alternatively two 
LEDs could be used in the above example, except that both LEDs blinking may 
prompt the driver to squeeze with both hands. 

\^The drivers response or level of alertness may be detected by measuring 
the response time of the driver, where the response time is measured as the time 
between illumination of an LED and a correct response with the hand or hands. 
In a case where an inappropriate response time is detected (potentially signalling 
driver fatigue or onset of driver fatigue) the system can verify the results and alert 
the driver. The system may also determine the accuracy of the driver's 
responses to ascertain the status of the driver's vigilance. 

A further example of means for testing the driver's response may Include 
means for flashing random numbers on the windscreen. The driver may be 
prompted to respond by squeezing the steering wheel a number of times as 
determined by the number flashed. The numbers may be flashed on the screen 
at different locations relative to the steering wheel with the position of the hands 
on the wheel responding to the side of the screen where the flashes were 
detected. This type of test should be conducted only when the driver is not 
turning, changing gear, braking or performing other critical driving functions. 

It Is desirable to ensure that the driver response tests are not anticipated 
by the driver to more accurately detect the driver's state of vigilance. It is of 
course also important that the selected method of testing driver response, does 
not in any way distract the driver or contribute to the driver's lapse in 
concentration. 
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The system may be built Into a vehicle sun visor as a visual touch screen 
display allowing a comprehensive visualisation of a drivers activity. The touch 
screen may Include a color display for displaying movement/pressure outputs 
associated with each sensor. A display of the status of a plurality of sensors may 
5 provide a visual Indication of a relaxed versus an active driver state. 

'i^j According to one aspect of the present Invention, there Is provided 
apparatus for determining a vigilance state of a subject such as a driver of a 
vehicle or the like, said apparatus including: 

' means for monitoring one or more physiological variables associated with 
10 said subject; 

means for deriving from said one or more variables data representing 
physiological states of said subject corresponding to the or each variable; and 

* means for determining from said data when the vigilance state of said 
subject is below a predetermined threshold. 
15 5^ According to a further aspect of the present invention, there is provided a 
method for determining a vigilance state of a subject such as a driver of a vehicle 
or the lil<e, said method including the steps of: 

^ monitoring one or more physiological variables associated with said 
subject; 

20 deriving from said one or more physiological variables data representing 

physiological states of said subject corresponding to the or each variable; and 

, determining from said data when the vigilance state of said subject is 
below a predetermined threshold. 

Preferred embodiments of the present invention will now be described with 
25 reference to the accompanying drawings wherein:- 

Fig. 1 shows a block diagram of a vigilance monitoring system according to 
the present invention; 

Fig. 2 shows a flow diagram of an algorithm for processi ng data from 
sensors associated with a vehicle and driven 



30 Fig. 3A shows a simplified block diagram of a system for cancelling 

environmental noise from driver interfaced sensors; 

Fig. SB shows waveforms associated with the system of Fig. 3A; 
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Fig. 4A shows a flow diagram of a movement processing algorithm 
according to the present invention; 

Fig. 4B shows examples of data reduction and syntactic signal processing 
associated with a sample signal waveform; 

Fig. 5 shows sample outputs of data following secondary analysis by the 
system of Fig. 4A; 

Fig. 6 shows an embodiment of steering wheels sensors; 
Fig. 7 shows a block diagram of a vigilance monitoring system utilizing 
video data; 

Fig. 8 shows a flow diagram of an algorithm suitable for processing video 

data; 

Figs. 9 and 10 show examples of data produced by the system of Figs 7 

and 8; 

Fig. 11 is a flow diagram of the main vigilance processing algorithm; 
Fig. 12 is a block diagram of a vehicle monitoring system according to the 
present invention; 

Fig. 13 shows one fonm of transducer for monitoring posture of a driver or 
equipment operator; 

Fig. 14 shows a block diagram of an embodiment of an anti snooze device 
according to the present invention; 

Fig. 15 shows a calibrate mode algorithm; and 

Fig. 16 shows a main relax detection algorithm. 
^ Referring to Fig. 1 , block 12 shows a plurality of sensors 1 to 1 1 associated 
with a vehicle and driver. The or each sensor may include piezoelectric or 
electrsstati^atedal^ucr^^ materiaT^ The material can be 

divided into plural sections of the driver's seat! for exampte. The various sensors 
are summarized below. 
1. Upper Driver seat sensor 

♦ Drivers seat top edge of upper section 

♦ Drivers seat centre of upper section 

♦ Drivers seat base of upper section 
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o 2. Lower Driver seat sensor 

♦ Drivers seat front edge of lower section / ^ 

♦ Drivers seat centre of lower section 

♦ Drivers seat rear of lower section 



3. Driver seat^belt sensor 

♦ Driver's seat-belt upper section 

♦ Driver's seat-belt lower section 

10 .4. Driver's head tilt per driver cap or similar sensor 

The driver's head tilt per driver cap or a device to clip over drivers ear or as part 
of driving goggles or glasses. These sensors can be, for example, positional 
switch devices. The output from these positional devices is amplified, filtered 
and finally data acquisitioned and analysed. This sensor device is designed to 
3 output a signal or digital data which changes state in accordance with the tilt of 
the driver's head. By calibration of the system in accordance with normal driving 
conditions this output can correlate the normal driving condition with the fatigued 
driver condition. 



5. Driver headband sensor 

The driver headband sensors can be» for example, a Capacitive Static Discharge 
l\/laterial (CSDM) or PVD Material (PVDM) that can be divided into the various 
sections (as listed below) of the driver's headband sensor. The output from the 
various sensors is amplified, filtered and finally data acquisitioned and analysed. 
The headband material can contain conductive sections designed to pick-up the 
patient's electro-encephalograph (EEG) signals. 

Driver headband sensor 

♦ Driver headband left frontal 

♦ Driver headband right frontal 

♦ Driver headband left eye 

♦ Driver headband right eye 
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EEG, EMG and EOG parameters monitored in critical driving conditions. 

In some ciitlcal applications of vigilance monitoring, such as pilots of aircraft, 
personnel responsible for navigating and controlling ships, drivers of road or rail 
transport or passenger vehicles, it can be appropriate to apply more 
comprehensive methods of vigilance monitoring. These more comprehensive 
monitoring techniques can include techniques for analysing the frequency 
composition of a subjects EEG physiological data. Half Period Amplitude analysis 
(AU patent 632932) or spectral analysis can be applied in order to determine if 
the subject is entering a trance or non-vigilant state or if the subject is becoming 
drowsy. This type of sleep staging can be derived in real time to facilitate 
detemiination of the subject's state of vigilance. If the subject is detected as 
being in a risk category the present system will alert the driver in order to prevent 
a potential vehicle accident due to the driver's lapse In concentration. 

One method of electrode attachment, but not limited to. could be the application 
of a headband by the driver where this head-band and/or chin-band could 
connect the EEG, EMG and EOG signals to the monitoring device for purpose oT 
analysing the signals for detennination of the subjects state of wakefulness. 

6. Driver Eye sensor 

Various techniques can be applied for the purpose of eye movement monitoring 
including: 

♦ Techniques based on reflected light. 

♦ Techniques based on electric skin potential. 

♦ Techniques based on Contact lenses 

♦ Techniques based on Limbus tracking 

♦ Techniques based on video imaging 

♦ Techniques based on Magnetic Induction 

♦ Driving goggles or glasses with infra-red detection capability for monitoring 
driver's eye movements, or EOG signal pick up via electrodes. 



Driver Eye Detection Sensor Types; 
♦ Driver's eyes left 
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♦ Driver's eyes right 

• sources of eye movements can include EOG electrodes, Infrared detection of 
eye movements, or video tracking and processing of eye movements. 

7. Driver steering wlieel sensor 

Ttie driver steering wheel or other steering device sensors can be, for example, a 
CSDM or PVD material that can be divided into the various sections (as listed 
below) of the driver's steering wheel or other steering device. The output from the 
various sensors is amplified, filtered, and finally data acquisitioned and analysed. 



Driver steering wlieel sensor types; 

♦ Drivers steering wheel top left section 

♦ Drivers steering wheel top right section 

♦ Drivers steering wheel bottom left section 
15 ♦ Drivers steering wheel bottom right section 

An alternative form of steering wheel sensor is shown in Fig. 6. 

8. Driver carpet region sensor 

20 The driver carpet sensors can be, for example, a Capacitive Static Discharge 
Material (CSDM) or PVD Material (PVDM) that can be divided into the various 
sections (as listed below) of the driver's carpet area. The output from the various 
sensors is amplified, filtered and finally data acquisitioned and analysed. 

25 9. Driver accelerator sensor 

The driver accelerator sensors can be, for example, a Capacitive Static Discharge 
Material (CSDM) or PVD Material (PVDM) that can be divided into the various 
sections (as listed below) of the accelerator pedal. The output from the various 
sensors is amplified, filtered and finally data acquisitioned and analysed. 



Driver accelerator pedal sensor types; 

♦ Drivers accelerator pedal top section 

♦ Drivers accelerator pedal center section 
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♦ Drivers accelerator pedal bottom section 
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1 0. Driver clutch pedal (where applicable) sensor 

The driver dutch sensors can be. for example, a Capacitive Static Discharge 
5 Material (CSDM) or PVD Material (PVDM) that can be divided into the various 
secttons (as listed below) of the driver's clutch pedal (where applicable). The 
output from the various sensors is amplified, filtered and finally data acquJsitioned 
and analysed. 

1 0 Driver clutch pedal sensor types; 

♦ Drivers clutch pedal (if applicable) top section: 

♦ Drivers clutch pedal (if applicable) center section 

♦ Drivers clutch pedal (if applicable) bottom section 

15 11. Driver brake pedal sensor 

The driver brake sensors can be. for example, a Capacitive Static Discharge 
Material (CSDM) or PVD Material (PVDM) that can be divided into the various 
sections (as listed below) of the brake pedal. The output from the various 
sensors is amplified, filtered and finally data acquisHioned and analysed 



20 



25 



30 



Brake Pedal sensor types; 

♦ Drivers t)rake pedal top section 

♦ Drivers brake pedal center section 

♦ Drivers brake pedal bottom section 

Other sensors are referred to in block 13. Including steering wheel 
movement and direction sensors and sensors for detecting environmental noise 
and vibrations. 

The outputs from the various sensors are amplified and filtered in block 14 
.n preparation for analog to digital conversion In block 15. The sensor signals are 
.nput in digital fbm) to block 16. Block 16 includes a central processing unit and 
one or more algorithms for processing the digital signals. Block 16 also makes 
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use of the vigilance processing algoritlinn(s) in block 17. The vigilance processing 
algorithm(s) in block 17 are adapted to determine the status of the driver state of 
vigilance, alertness or wakefulness. This status may be expressed as a vigilance 
fector (0-100%). Upon detecting a vigilance factor which is below an acceptable 
5 threshold, the central processing unit may alert the driver of the vehicle and/or 
other vehicles. The driver alert means may include: 
To alert external drivers; 

♦ Flashing Hazard Lights 

♦ Sounding of siren 

10 

Internal vehicle driver alert systems', 

♦ Scent sprays which are designed to activate the drivers vigilance state 

♦ Vibration modulation for driver- can include vibration of steering wheel or floor 
area to alert driver , 

15 ♦ Vibration modulation for driver seat-belt 

♦ Vibration modulation for driver steering wheel 

♦ Audible alarm system at frequencies and durations or sequence of durattons 
as tested be most effective in alerting the driver 

♦ Cabin bright lights designed to avoid driving hazard but tested for improving 
20 driver vigilance 

Upon detecting a vigilance factor whtoh is below an acceptable threshold, 
the central processing unit may Intervene in the control of the vehicle. Vehicle 
intervention may enable the vehicle to be brought Into a safe or safer status. 
25 Vehicle intervention may include speed restriction or reduction or complete 
removal of fuel supply. In some circumstances the accelerator pedal may need to 
be disabled, for example when a driver has his foot depressed on the accelerator 
pedal and is in an unsafe or fatigued state. 

30 Where a driver is detected as Ignoring or not responding to response requests or 
appropriate acknowledgement that the driver is in a vigilant state, the vehicle may 
have its horn or hazard flashing lights activated to warn other drivers, and/or have 
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its fuel injection de-activated, and/or speed reduced by gentle and controlled safe 
braking. 

Where a driver is detected as suffering from fatigue and is not responding to 
5 response tests, the vehicle may have its fuel supply reduced, and/or its speed 
reduced by gentle and controlled safe braking, to a safe cruising speed. The 
driver may then be prompted again before the vehicle undergoes further 
intervention. 

Another option for vehicle intervention is to provide a form of ignition 
10 override, as used in some alcohol based systems. In this type of system the 
vehicle ignition or starting process may be inhibited by an Inappropriate driver 
state which In the present case may be drowsiness or excessive fatigue. 

In many modem vehicles vehicle intervention options may be instigated by 
an onboard computer or electronic interface eg. by communication with the speed 
1 5 controller or fuel Injection logic. The computer system, may include intelligence to 
arbitrate the most appropriate intervention sequence or process to minimize risk 
to the vehicle driver or its passengers. 

Fig. 2 shows a ftow diagram of an algorithm for processing data from 
sensors associated with the vehicle and driver. Block 20 shows a plurality of 
anxjws on the left representing data Inputs from various sensors associated with a 
vehicle, following conversion to digital data. The digital data is input to block 21 
which determines whether the data confomis to valid amplitude thresholds stored 
in block 22. Signals beyond the thresholds are classified as noise or artefact and 
are ignored. The data is then input to block 23 which detects whether the data 
conforms to valid time duration thresholds stored in block 24. Signals beyond the 
thresholds are classified as invalid and are ignored. The thresholds stored in 
blocks 22 and 24 are. for the purpose of the present embodiment, determined 
empirically from experimental trials. The data is then input to block 25 for signal 
compression. The role of block 25 Is to simplify further processing by presenting 
the data in a minimized form. This is done by syntactic processing whereby main 
data points only of the signals such as various peaks, troughs and zero crossings 
or central points defining peaks of the signals are presented for further 
processing. The data is then input to block 26 where it is categorized and 
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summarized in tems of amplitude or power range, number of movements per 
second or other epoch, and phase relationships between the signals. The data 
may be displayed on tabular or graphical form and/or may be subjected to further 
automated processing to determine vigilance status. 

5 Fig. 3A shows a blocic diagram of a system for removing environmental 

noise from driver interfaced sensors. Block 30 represents various sensors for 
monitoring driver movements and block 31 represents sensors for monitoring 
environmental vibration and noise and vehicle artefacts. 

Blocks 32 and 33 represent circuits for amplifying and filtering signals from 

10 blocks 30 and 31 respectively. Bk>ck 34 represents analogue to digital converters 
for converting the signals from btocks 32 and 33 into digital form for processing 
via the digital signal processor in block 35. Block 35 includes an algorithm for 
performing signal cancellation as illustrated in Fig. 3B. 

In Fig. 3B waveform A represents a signal from a driver Interfaced sensor 

15 or sensors (Block 30 of Fig. 3A). Waveform B represents a signal from a sensor 
or sensors associated with the vehicle engine and road noise pickup locations 
(Block 31 of Fig. 3A). Wavefomi C represents a signal after It is processed by 
Block 35. It may be seen that the signal represented by waveform C is obtained 
by cancelling or subtracting the signal represented by waveform B from the signal 

20 represented by waveform A. The signal represented by waveform C is a true or 
valid nfK>vement signal which is not corrupted by environmental noise. 

Figure 4A shows a flow diagram of a movement processing algorithm 
according to the present invention. Refening to Fig. 4A^ signals from sensors 1 to 
11 shown in block 12 of Figure 1 are filtered, then referenced to period and 

25 amplitude threshold values before being converted to syntactic data. The 
syntactic data is correlated for determinatton of certain combinations of sensor 
movement signals indicating that the driver is in a vigilant or wakeful state. When 
a sensor signal or any combinatton of sensor signals are analysed as being void 
of subject movement, this may be interpreted as an indication the driver is 

30 suspected of being in a non-vigilant or fatigued state. Analysis of the fatigued 
state is determined by certain expected patterns from the various sensor signals. 
Such patterns include very litOe movement from the steering wheel and very litOe 
nfu>vement from the seat sensors, indicating that the driver may be excessively 



wo 00/44580 „^..,. 

PCT/AU99/01166 

-22- 

relaxed and subject to fatigue, or at risk of fatigue on-set. The functions of blocks 
40 to 61 are as follows: 
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BLOCK 40 

Figure 1 shows how the analog signals from sensors 1 to 11 are: converted to a 
digital signal (figure 1. block 15); input to the central processing unit (figure 1, 
block 16): and processed by a vigilance processing algorithm (figure 1, block 17). 
The start of the algorithm in figure 4A represents the start of a process, which is 
repeated many times for each Input sensor 1 to 1 1 (Fig. 4A shows the process for 
sensors 1. 2. 3). This process analyses data from each Input sensor for the 
purpose of final detemnination of the driver's vigilance state, and whether this 
state warrants an alami alert In order to assist In preventing a potential accident. 

BLOCKS 41 TO 46 

Signal A/D Data Output. The analog signal from each sensor is amplified, filtered 
and then converted to a digital signal in preparation for signal processing. 
Variables A,C,E-U 

Variables A.C.E.-U provide to the processing algorithms threshold amplitude and 
period values to allow sensor signal data reductions to be determined and to 
allow data reduction and syntactic signal processing. The variables (A.C.E - U) 
are detemiined via controlled studies from experimental and research data. 

Fig. 48 shows examples of: (1) signals components which are ignored due to 
being below a minimum amplitude threshold. (2) syntactic data where the signal is 
represented by troughs and peaks of the signal, and (3) high frequency 
component being ignored due to being below a minimum period threshold. The 
latter recognizes relatively lower frequencies which are typically due to driver 
movements. 

Inputs from sensors 4 to 11, subject to system configuration 

Input from each of the vehicles sensors is amplified, filtered and then analog to 
digital converted, in preparation for signal processing. This is performed by 
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blocks similar to blocks 41 to 46. inputs from more tlian 1 1 sensors can be 
catered for if required. 

BLOCK 47 
5 A,C,E,-U 

Variable data via default table (as detemiined by clinical data and/or neuro node 
self learning and adjustment), resulting from customisation to specific subject's 
driving cliaracterlstics and system adaptation. 

Variables: B,D,F,-V . By comparing tlie sensor data to various amplitude 
10 thresholds and pulse periods, It is possible to ignore data that is likely to be noise 
or artefact and include data that is distinguishable as movement data from a 
driver. The movement data is distinguished by measuring the amplitude and 
period characteristics of the sensor signal. Movement data is also distinguished 
by comparing signal patterns and characteristics of sensors to patterns and 
15 characteristics of typical driver's movements (as determined by comparative date 
used for correlating against current date, this date being derived from system self- 
leaming and/or calibration processes.) 
BLOCK 48 

Is peak to peak amplKude of sensor output greater than threshold variable A 7 
20 Retein time reference and value of each signal excursion of input sensor 
exceeding amplitude reference A. 

BLOCK 49 

Is peak to peak amplitude of sensor output greater than threshold variable C 7 
25 Retein time reference and value of each signal excursion of input sensor 
exceeding amplitude reference 0. 

BLOCK 50 

Is peak to peak amplitude of sensor output greater than threshold variable E 7 
30 Retein time reference and value of each signal excursion of input sensor 
exceeding amplitude reference E. 
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BLOCK 51 

Is peak to peak amplitude of sensor output greater than threshold variable B ? 
Retain time reference and value of each signal excursion of input sensor 
exceeding amplitude reference B. 

BLOCK 52 

Is peak to peak amplitude of sensor output greater than threshold variable D ? 
Retain time reference and value of each signal excursion of input sensor 
exceeding amplitude reference D. 

BLOCK 53 

Is peak to peak amplitude of sensor output greater than threshold variable F ? 
Retain time reference and value of each signal excursion of input sensor 
exceeding amplitude reference F. 

Storage of input sensors period and amplitude with time reference 

The syntactic data from the full range of sensors Is stored in random access 
memory for the purpose of processing and determination of a subject's vigilant 
state. The syntactic data is also archived to allow post analysis report and 
validation or review of driver fatigue and performance. This can be particularly 
useful where truck drivers and other critical transport or passenger drivers are 
required to be checked for performance and vigilance compliance. 

BLOCK 54 

Longer-term data storage is designed to log the driver's movement data from 
each of the sensors. This stored data can be accessed at a later stage in order to 
review the driver's perfomiance history in regards to movement analysis and 
subsequent vigilance. 

BLOCK 55 

Short term direct access storage used for storing parameters such as the past 10 
minutes of syntactic data for each sensor channel, in order to correlate the 
various data from each sensor or channel and compare this data combination to 
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pre-defined sets of rules designed to describe combinations of sensor outputs 
which are typical of driver fatigue conditions. 

BLOCK 56 

5 Store syntactic representation of sensor signal exceeding threshold A and B, with 
timer reference, amplitude and pulse width. 

BLOCK 57 

Store syntactic representation of sensor signal exceeding threshold C and D. with 
1 0 timer reference, amplitude and pulse width. 

BLOCK 58 

Store syntactic representation of sensor signal exceeding threshold E and F, with 
timer reference, amplitude and pulse width. 

15 

BLOCK 59 

Driver specific profile and calibration data can be stored for later correlation 
reference. By correlating with various thresholds or reference conditions the 
system is able to determine interaction to sensors when a particular driver's 
20 conditions is similar to pre-stored reference characteristics. This comparative 
data Is stored as data In loolc up tat>les. The data can consist of firequency and/or 
amplitude characteristics for a range of driver states or alternatively the data can 
consist of samples of data (with acceptable variations to the samples of data) that 
exist for a range of driver states. 

25 

BLOCK 60 

Vehicle output signals. These include steering wheel movements, direction of 
steering wheel movemente, speed of vehicle, change of speed of vehicle, engine 
vibration and noise, road vibration and noise. 

30 

By processing driver steering wheel adjustments and comparing these 
adjustments with the various sensor signals and correlation of various sensor 
signals. It is possible to determine the probability that the driver is in a state of 
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fatigue and the degree of driver fatigue. The vehicle signals are also analysed in 
order to assist In noise cancellation (le vehicle noise as opposed to driver 
movement) and more accurate identification of valid driver movements). 

5 BLOCK 61 

Con-elate all channels of sensor activity and determine if driver fatigue is a 
probability and what level of driver fatigue Is detected. Look up table of specific 
driver calibration values and reference states is used to detennine actual driver 
state and level of fatigue of driver, along with probability of data accuracy. 
10 Standard reference data tables and default values are also used for determination 
of driver fatigue. See sample R&K style histograms, movement histograms and 
tables showing weighted value of each sensor and actual movement detection 
from each sensor to detefrmine fatigue probability as a function of movement 
detection with appropriate weighting. 
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Fig. 5 shows typical samples of pnacessed data following secondary 
analysis for sensor signals 1 to 4. The data shows in graphical form the number 
of valid movements detected for each sensors 1 to 4 during successive time 

inten^als n, n+1, n+2 Tertiary analysis may be performed on this data which 

would allow simple to view correlation between the various sensors. The samples 
shown in Fig. 5 demonstrate an example (dotted line) where the various sensors 
all experience obvious movement detection. 

The steering wheel sensors shown in Fig. 6 are divided Into eight sections 
as follows: 



Top 62. top left 63. top right 64. left 65. right 66. bottom left 67. bottom 
right 68 and bottom 69. 

Sensors 62-69 are linked via eight cables to output pins 1 to 8 respectively. 
A common connection to each sensor is linked by cables to output pin 9. 
Alternative configurations are possible with more or less sensors and with the 
option of sensor an-ays on both the upper and lower surfaces of the steering 
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wheei grip surface. The outputs represented by pins 1 to 9 are connected to 
analogue signal conditioning circuits and via analogue to digital converters to 
digital signal processing circuits as described above. 

It is desirable to measure pressure of a driver's hand or hands on the 
5 steering wheel at all times. The pressure may be compared to previous values 
and/or calibrated values to determine whetiier a pattem of increased or 
decreased pressure reflects driver fetigue onset. 

If the driver's state of consciousness or concentration changes due to 
fatigue onset or the lil<e, the system may calculate and deduce an appropriate 

10 point at which the driver should be alerted. The appropriate point may be 
determined from a combination of pre-calibrated data for a specific driver and/or 
pre-programmed patterns, states or trends in the data Including relative and 
absolute pressure values obtained from a set or subset of vehicle sensors. 

Fig. 7 shows a block diagram of a vigilance monitoring system utilizing 

15 video data. Block 70 represents a video CCD (charge coupled device) camera 
which may be located on the driver's visor, dash-board or other suitable location 
to enable video monitoring of the driver's eyes. An infra-red lens may be utilized 
to facilitate reliable night video monitoring capability. The output of the video 
camera Is passed to block 71. Block 71 is an analog to digital converter for 

20 digitizing the video signal prior to processing via block 72. Block 72 is a central 
processing unit and includes a video processing algorithm. The video processing 
algorithm has eye recognition software designed to identify eyes in contrast to 
other parts of the drivers face. Eyes are detected using special processing 
software that allows the driver's eyes to be analysed. This analysis includes 

25 determining the area of the eye's opening and correlating the eye's opening area 
to previous similar measurements. In this way eye processing can detemnine 
whether a driver's eyes are remaining open as would be expected in an alert 
state or whether the current eye opening of the driver is relatively less (when 
compared to eariier eye opening measurements). Rates or degrees of eye 

30 closure are able to be detected and continually monitored in tills manner. 

The video processing algorithm also detects blink rate and possibly eye 
movements to determine whether the drivers eyes appear to be alert or possibly 
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fixed in a dangerous "trance state" as may be apparent during lapses of driver 
vigilance. Block 73 represents outputs of block 72 Including 

eyes blink rate 

eyes closure, calculated as a percentage ratio of current eyes open area to 
previously calculated maximal eyes open area. 

eyes focus factor, determined by measuring number of eye movements per 
second, extent of eye movements (ie small eye movements or larger eye 
movement deflections) 

the nature of eye movements can reflect appropriate patterns of movement 
of a driver's eyes such as focus on sections of the road for an appropriate 
time as well as Inappropriate patterns of movement associated with fatigue 
or lack of vigilance 

type of eye movements, ie vertical, horizontal, stare 

The above measures may be gauged against actual trials in order to determine 
relevant indices that correlate to a driver's fatigued state. 

Fig. 8 shows a flow diagram of an algorithm suitable for processing video 
data. The functions of blocks 80 to 94 are as follows: 

BLOCK 80 

START 

BLOCK 81 

CAPTURE EYE VIDEO DATA- Capture current video frame. Digitise video frame 
of subject s eyes. Eye data can be captured via one or more of the following 
means: 

CCD video camera, Electro-oculogram data capture means via subject wom 
headband, direct electrode attachment, driver glasses, head-cap or movement 
sensors, infrared or other light beam detection means. 
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BLOCK 82 

APPLY EYE DATA PROCESSING AND DETERAAINE LEFT & RIGHT EYE 
OPENING AREA AND BLINK EVENTS. 

Apply edge detectbn. signal contrast variation and shape recognition, amongst 
other processing techniques to determine the bord&r of the subject's eye lids. 
Determine area of each of the subject's eye openings, height of each eye 
opening, blinl< events for each eye, blinl^ rate and time reference associated with 
each btinic event. 

BLOCK 83 

CORRELATE CURRENT AND PAST VIDEO CAPTURED EYE MOVEMENT 
DATA 

Correlate cun^nt eye position data with previous position eye data. Review eye 
position trend data and determine trends^ and patterns of eye movements that 
indicate on-set of or driver fatigue state. Patterns include: 

- states of staring or trance lilce states Indicating loss of road concentration. 

- slowly rolling eye movements (typical of sleep onset). 

- eye focus directions and association of these directions with driver fatigue 

Process digitised video frame and detect subject's left and right eye movement 
patterns and activity of eyes and association of this activity with driver fatigue. 

Compare cun-ent blink rates, past bllnK rates and look-up table blink rate 
characteristics, thresholds for various fatigue on-set and fatigue blink rates and 
blink characteristics associated with various driver states. 

Compare cun^nt eye opening area with thresholds for fatigue and fatigue on-set 
conditions to determine vigilant driver eye opening status versus fatigued driver 
eye opening status. 

BLOCK 84 

Look up table with characteristic patterns of; 

- eye movements and threshold data for fatigued versus vigilant subjects. 
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- Blink rate typical thresholds and characteristics 

- Eye opening typical and default thresholds 

- Eye nnovennent typical and default characteristics for driver fatigue on-set. 
BLOCK 85 

Store subject's left & right eye opening area, eye opening height, blink rates, eye 
position and eye movements together with time reference. 

BLOCK 86 

Calibration data derived from subject and vehicle calibration procedures. 

- determination of fatigue on-set blink rates thresholds. 

- Determination of eye opening fatigue on-set thresholds. 

- Determination of eye position, movement characteristics and activity 
characteristics for fatigue on-set thresholds. 

- EOG patterns for wake, drive activity, fatigue on-set, fatigue, 

- Trance and hypnotic EOG eye characteristics. 

BLOCK 87 

Fatigue threshold time period variable X set from default values, subject 
calibration or system se|f-leaming/calculation. 

BLOCK 88 

Is mean of eye opening area below "fatigue mean eye opening threshold X "? 
BLOCK 89 

Fatigue threshold time period variable Y set from default values, subject 
calibration or system self-leaming/calculation. 



BLOCK 90 

Is time duration below mean eye opening fatigue threshold (X) greater than Y ? 
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BLOCK 91 

Blink rate fatigue cliaracteristics set from default values, subject calibration or 
system self-leaming/calculation. 

5 BLOCK 92 

Does b\mk rate and characteristics comply with fatigue blink rate characteristics 7 
BLOCK 93 

Apply eye data processing and determine left & right opening area and blink 
10 events. 

Correlate current and past video captured eye movement data. j 

Detection of fatigue eye opening on-set and detection of fatigue blink rate on-set 

15 

BLOCK 94 

Eye movement fatigue determination diagram. 

Fig. 9 and 10 show examples of eye opening and eye position data produced by 
20 the system of Figs. 7 and 8. 

Fig. 1 1 is a flow chart of the main vigilance processing algorithm. Jhe functions 
of blocks 95 to 99 are as follows: 

25 BLOCK 95 

MAIN VIGILANCE PROCESSING ALGORITHM 

♦ Vigilance Movement Processing Algorithm, (see figure 4A) 

i 

♦ Vigilance Eye Status Processing Algorithm. 

30 

Probability of Driver Fatigue and Degree of Vigilance Determination Algorfthm 
(correlates subject Movement Status and Eye Processing Status). = 
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BLOCK 97 

Eye Status Vigilance factor 0-100 %. 
BLOCK 98 

Movement Vigilance Factor 

0- 100 % -displayed as bar graph, meter or other means. 
BLOCK 99 

Vigilance probability Factor 0-100 % 

Fig. 12 is a block diagram of a vehicle monitoring system according to the 
present invention. Fig. 12 is an overview of a system which utilizes many of the 
features discussed herein. The functions of blocks 100 to 1 18 are as follows: 

BLOCK 100 

Driver EEG sensors- direct attach electrode, headband, wireless electrode, driver 
cap and other EEG signal pick-up means. 

BLOCK 101 

Driver EEG sensors- direct attach electrode, headband, wireless electrode, driver 
cap and other EEG signal pickup means. 

BLOCK 102 

Driver Motion. Movement and Physiological Parameter sensors. 

BLOCK 103 

Driver Eye movement 

Detection via electrode, driver glasses/goggles, infrared or other light beam 
means of tracking detection or other means. 
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LED indicator display panel. 
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BLOCK 104 

Vehicle status interface; speed, direction, accelerator position, break position, 
indicators, lights amorigst other vehicle status data. 

5 BLOCK 105 

In phase signal detection and processing. Applies processing which determines 
patterns of in-phase signal occurrence and associates these with driver or 
background noise as originating source. 

10 BLOCK 106 

Anti-phase signal detection and processing. Applies processing which determines 
patterns of anti-phase signal occurrence and associates these with driver or 
background noise as originating source. 



15 BLOCK 107 

Vehicle Background Noise Processing Algorithm. 

Vehicle background and Environmental Noise Sensors to allow noise 
cancellation, filtering and reduction. 

20 These sensors Include microphone and vibration sensors located at strategic 
positions in order to pick up background vehicle noise such as road noise and 
engine noise. Fourier transform and frequency analysis of background noise 
assists in selection of digital filtering characteristics to most effectively minimise 
vehicle environmental noise and assist in distinguishing driver related fatigue 

25 monitoring signals. System will continually "self-leam" various vehicle 
background and threshold noise levels, frequency and other characteristics in 
order to determine changing vehicle noise conditions and subsequent noise 
cancellation or capability to ignore unwanted vehicle noise while processing "real" 
driver movenrient and physiological signals and subsequent fatigue status. 

30 Artificial intelligence; 

Signal characteristics as generated by a range of varying road conditions can be 
programmed into the system. The input data relating to various road conditions 
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thereby provides a means to further distinguish wanted driver related signals from 
unwanted background noise signals. 

BLOCK 108 

5 Driver EEG sensors- direct attach electrode,Algorlthm 
BLOCK 109 

Driver EEG sensors- direct attach electrode, algorithm 

10 BLOCK 110 

Driver Motion. Movement, Physiology algorithm 

BLOCK 111 

Driver Eye Movement Detection Algorithm 

15 

BLOCK 112 

Vehicle status interface Algorithm 
BLOCK 113 

20 Driver Fatigue Processing Algorithm. Conrelation with previous driver fatigue 
conditions together with comparison of outputs for each of above listed fatigue 
algorithms (Driver EEG, motion, eye, vehicle status). 

BLOCK 114 

25 Driver vigilance interactive response testing. 
BLOCK 115 

Driver alert and alarm systems for re-instatement of vigilance. 
30 BLOCK 116 

Driver vehicle car intervention to reduce or limit speed and other means of 
increasing vehicle safety and reducing vulnerability to driver fatigue status. 
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BLOCK117 

Vehicle fatigue display systems for displaying to the driver the current fatigue 
status or early warning Indicators of fatigue status. 

5 BLOCK 118 

System communication storage and printing peripheral interface. Data storage, 
reporting processing, reporting print interface, wireless and wire connected 
interfaces, for real-time or post communication of fatigue data and fatigue status 
Information. System can include GSM. cellular phone, satellite or other means of 
10 moving vehicle tracking and data exchange in real-time or at any required later 
stage. This Information transfer can be an effective means for trucks and other 
vehicles to have their driver status processed and reviewed, as appropriate and 
as required. 

15 Fig. 13 shows one iorm of transducer for monitoring posture of a driver or 

equipment operator. Fig. 13 shows a webbed structure comprising strips or 
elements of flexible PVDF or Piezo material separated by flexible insulation 
material terminated at A, B, C, D, E. F, G and H. Output signals from the 
respective strips are buffored, amplified, filtered and then analog to digital 

20 converted to data. This data may be processed to determine an actual position of 
pressure applied to the at)ove structure.. By analysing the two main co-ordinates 
and the amplitudes of signals associated with those co-ondlnates. the exact 
position of pressure applied by the vehicle driver or equipment operator may be 
determined. 

25 The position where greatest pressure is applied Is defined by the 

Intersection of web strip pairs (eg. Band F) which produce the greatest signal 
amplitude. The position may be described by coordinates reflecting the web strip 
pairs (eg. B,F) which produce the greatest signal amplitude. The above 
transducer may be used In conjunction with the movement sensors described 

30 herein to provide a further layer of positional information relating to applied 
pressure for each sensor. This information may be Important in circumstances 
where a driver's pressure to the steering wheel or the driver's pattern of hand 
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placement (with respective applied pressure) varies In accordance with alertness 
and drowsiness. 

The posture of the driver or equipment operator may be monitored, stored, 
correlated with various threshold states and/or displayed in meaningful graphic or 
numerical fomi. The threshold states may be derived by way of calibration for 
each specific driver's posture profile under various states of fatigue and/or stress 
states and conditions. 

The anti-snooze device shown in figure 14 Includes sensors (block 120) 
connected to an acquisition and processing means (block 121). Block 122 
includes monitoring means designed to amplify, filter and digital to analog convert 
driver sensor signals in preparation for digital signal processing. The digital signal 
pnacessing means (block 121) includes a calibration algorithm as shown in figure 
1 5 and a main relax detection algorithm as shown in figure 1 6. 

The driver can select the relax calibration function, then take on the driving 
posture that would most closely represents a relaxed or possibly fatigued driving 
state and the system will then monitor and store the minimum threshold of driver 
activity over a period of approximately but not limited to 10 seconds, as a relaxed 
driver reference level. 

The driver can select an active calibration function, then take on the driving 
posture that would most closely represents normal driving state and the system 
will then monitor and store the minimum threshold of driver activity over a period 
of approximately but not limited to 10 seconds, as an active driver reference level. 

The relaxed and active driver reference levels stored in the system may be 
displayed on the visual touch screen display for various sensors. The system 
may perfomn a validation function by replaying the drivers relaxed and active 
reference levels on the touch screen. This allows easy comparison to be made 
with actual sensor levels when the driver adopts postures representing 
normal/fatigued states and serves to validate the correctness of the stored 
reference levels. 

The driver can also select a sensitivity function which may determine how 
close to the relaxed level the driver needs to be before the anti-snooze system 
alerts the driver. By viewing the anti-snooze device screen the driver can relax or 
adopt normal vigilant driving posture and adjust sensitivity control so that the anti- 
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snooze device appears to track and detect the drivers relaxed state. The anti- 
snooze device has the ability to act as a self warning aid by simply alerting the 
driver when his posture or driving vigilance is deteriorating. If, for example, a 
driver's steering wheel grip erodes or undergoes fatigue, the anti-snooze system 
5 can be calibrated to detect this condition and alert the driver. 

It Is possible for the driver to have calibration data determined by an off- 
road simulator that more accurately defines the characteristics of each specific 
drivers activity variations and physiological variations during dangerously relaxed 
or fatigued driving conditions. The calibration data can be up-loaded to the anti- 

10 snooze device to provide more accurate relaxed and active reference levels. The 
calibration data may also provide more accurate means of determining the 
relative effect that each Individual sensor has during a drivers transition from 
active and alert to drowsy and fatigued. The effects of each sensor may be 
recorded and this data may assist In rrK>re accurate anti-snooze detection. 

15 During calibration modes the system may detect the drivers hand 

pressures via the steering wheel sensors, the drivers respiration and ECG via the 
seatbeit sensors, and the drivers posture and movement via the seat sensors. 

The anti-snooze system may continually monitor and average the signal 
amplitudes of all sensors, while comparing the current levels of sensor amplitude 

20 with the calibrated levels. The system may also compare current movement 
sensor patterns to reference data. This reference data can represerit certain 
threshold levels calibrated to each indi>ndual driver or general i|eference 

1 

conditions. The various sensors may be weighted in accordance yJ^h their 

respective importance in determining whether a driver's current state of Activity is 
25 below the threshold or appropriately close to the relaxed mode Calibrated 

reference level to warrant that the driver be alerted. | 

If the driver Is detected as. being within the range of sensor aniplrtijdes and 

activity to warrant being alerted, the anti-snooze device can restrict the jspeed of 

the vehicle or slowly bring the vehicle to a stand still in order to rejduce the 
30 likelihood of an accident. This ability to restrict the vehicle's speed could be 

overridden by the driver as is possible in "auto-cruise" devices currently available 

on many vehicles. 
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The techniques and methodologies may Include relatively complex 
neurological waveform analysis techniques, video tracking of driver eye motions, 
sophisticated noise cancellation and simpler driver interactive processes such as 
sensitizing the steering wheel, seat-belt, gear-stick and other driver cabin regions. 

One application for the present invention may include a truck driver 
vigilance nDonitoring (TDVM) system. This system may be designed around the 
"dead-man- handle concept as applied successfully in trains. A variation of this 
system may provide visual cues and driver vigilance response testing. 

The TDVM system may Include pre-programmed Light Emitting Diode 
(LED) displays to be activated in various sequences and at various frequencies 
and durations. The truck driver can be visually prompted by way of these LEDS 
to press the steering wheel according to whether the left or right or both LEDS are 
flashed. The response time and accuracy of the driver's response to the prompts 
may be measured and relayed back to a remote monitoring control station. 

Various drivers will have calibrated "vigilant response times and accuracy 
levels" which can be compared to actual cun-ent response times. Where 
appropriate, an alarm can be activated, if the response times Indicate fatigue on- 
set or a potentially dangerous state. 

The sequences and durations can be validated in accordance with clinical 
trials to provide an effective method of vigilance detection. Sequences and 
patterns of visual truck cabin prompts can be established to minimize driver 
conditioning. Frequency of vigilance test prompts can be detemiined in 
accordance with requirements as detemiined via field studies. 

Safety considerations to avoid driver distraction by the proposed 
monitoring system may be implemented. Techniques such as utilization of "busy- 
response prompts especially designed within the system to alert the monitoring 
control unit that the driver Is vigilant but unable to respond at the time due to 
driving demands. 

The TDVM system may include the following components: 

1. Analysis Softwam This software may include a processing algorithm(s) 
designed to evaluate various driver prompts and response times. Evaluation of 
these response times may produce a probability factor associated with driver 
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vigilance for each specific driver. Analysis capability of driver response times 
may be an important element of the system. Accuracy of vigilance probability 
outcome, clinical analysis and scientific validation associated with this process 
may determine effectiveness of the monitoring system. 

5 

2. Truck'CabIn steerlno^heai phvsloloaical movement transducer. 

This device may adapt to the truck steering wheel and provide output signals 
subject to a particular zone of the steering wheel, which has been activated by 
applying various degrees of pressure to the steering wheel. 

10 

3. Controller Unit & Monttorino bevice fCU&MD^. 

This devtoe may provide a communication link and data management for 
interfacing the truck's CU&MD to a renrwtely located monitoring statton. 
This device may also provMe the transducer interface and transducer signal 
15 recording and detection capabilities. 

This device may also output control to the driver indicator LEDS and record and 
transmit vigilance response times to the remote monitoring station. 

4. Vigilance LED disoiav. 

20 This device may be interfaced to the CU&MD unit and may provide visual 
response prompt to the truck driver. 

5. Remote recordlno. monitorino and analysis system. 

This system may facHitate a remote operators visual alarms when vigilance 
25 response times are outside acceptable thresholds. 

This system may also provkJe communication links to the truck. 

This system may also provide analysis and system reporting to allow real-time 

tracking of vigilance perfonriance and vigilance alarm status. 

Finally, it is to be understood that various alterations, modifications and/or 
30 additions may be introduced into the constructions and arrangements of parts 

previously described without departing from the spirit or ambit of the invention. 
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CLAIMS 

1 . Apparatus for determining a vigilance state of a subject such as a driver of 
a vehicle or the like, said apparatus including: 

means for monitoring one or more physiological variables associated with 
said subject: 

means for deriving from said one or more variables data representing 
physiological states of said subject corresponding to the or each variable; and 

means for determining from said data when the vigilance state of said 
subject is below a predetermined threshold. 

2. Apparatus according to claim 1 wherein said physiological variables 
include body movement and/or position. EEG, EMG & EOG signals, electrical 
skin resistance and eye and head movement and/or position. 

3. Apparatus according to claim 1 or claim 2 wherein said monitoring means 
includes at least one sensor placed on the subject or a zone associated with the 
vehicle. 



4. Apparatus according to claim 3 wherein the or each sensor includes CSD 
or PVDF material. 

5. Apparatus according to claim 4 wherein the sensor material is placed on a 
steering wheel, seat, seat belt and/or pedals of the vehicle. 

6. Apparatus according to claim 5 wherein the pedals Include an accelerator, 
clutch and brake pedals of the vehicle. 

7. Apparatus according to claim 1 wherein at least one of said means for 
deriving and said means for determining is provided via digital processing means. 

8. Apparatus according to any one of the preceding claims wherein said 
determining means includes a vigilance algorithm. 
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9. Apparatus according to claim 8 wherein said vigilance algorithm is adapted 
to conrelate said data and/or apply combinational logic to said data to detect 
patterns In said data which are associated with a vigilance state of the subject 
that Is below said predetermined threshold. 

10. Aisparatus according to claim 9 wherein said algorithm Incorporates one or 
more look up tables Including reference movement data and default states 
associated with vigilance states that are above and below said predetemilned 
threshold. 

11. Apparatus according to claim 8 wherein said algorithm is adapted to 
detemiine a vigilance probability factor as a function of weighted movement data 
values. 

12. Apparatus according to any one of the preceding claims including means 
for intervening with the control of the vehicle and/or alerting the subject, in the 
event that the vigilance state of said subject Is below said predetennined 
threshold. 

13. Apparatus according to claim 12 wherein said means for Intervening 
includes controlled application of brakes, cutting-off fuel supply and/or disabling 
the accelerator pedal of said vehkde. 

14. Apparatus according to daim 12 wherein said means for alerting includes 
at least one of a spray designed to stimulate the subject, vibrating a steering 
wheel, seat belt or floor area associated with the vehicle; an audible alarm and/or 
switching of bright lights In the cabin of the vehicle, and flashing hazanJ tights 
and/or sounding of an audible alarm such as siren or hom. 

15. Apparatus according to daim 12 wherein said means for Intervening is 
interfaced to a remote monitoring station via a v«dreless communication system. 



16. A method for determining a vigilance state of a subject such as a driver of 
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a vehicle or the like, said method including the steps of: 

monitoring one or more physiological variables associated with said 
subject; 

deriving from said one or more physiological variables data representing 
5 physiological states of said subject corresponding to the or each variable; and 

determining from said data when the vigilance state of said subject is 
below a predetermined threshold. 

17. A method according to claim 16 wherein said physiological variables 
10 include body movement and/or position, EEG, EMG & EOG signals, electrical 

skin resistance and eye and head movement and/or position. 

18. A method according to claim 16 or claim 17 wherein said step of 
monitoring includes placing at least one sensor on the subject or a zone 

1 5 associated with the vehicle. 

19. A method according to claim 18 wherein the or each sensor includes CSD 
or PVDF material. 

20 20. A method according to claim 19 wherein the sensor material is placed on a 
steering wheel, seat, seat belt and/or pedals of the vehicle. 

21. A method according to claim 20 wherein the pedals include an accelerator, 
clutch and brake pedals of the vehicle. 

25 

22. A method according to claim 16 wherein at least one of said step of 
deriving and said step of determining is performed via digital processing means. 

23. A method according to any one of claims 16 to 22 wherein said step of 
30 determining is performed via a vigilance algorithm. 



24. A method according to claim 23 wherein said vigilance algorithm is 
adapted to correlate said data and/or apply combinational logic to said data to 
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detect patterns in said data which are associated with a vigilance state of the 
subject that is below said predetermined threshold. 

25. A method according to daim 24 wherein said algorithm Incorporates one or 
5 more look up tables Including reference movement data and defeutt states 

associated with vigilance states that are above and below said predetermined 
threshold. 

26. A method according to claim 23 wherein said algorithm is adapted to 
10 determine a vigilance probability factor as a function of weighted movement data 

values. 

27. A method according to any one of claims 16 to 26 Including the step of 
intervening with the control of the vehicle and/or alerting the subject, In the event 

1 5 that the vigilance state of said subject is below said predetermined threshold. 

28. A method according to claim 27 wherein the step of intervening includes 
controlled application of braises, cutting-off fuel supply and/or disabling the 
accelerator pedal of said vehide. 

20 

29. A method according to daim 27 wherein the step of alerting indudes at 
least one of applying a spray designed to stimulate the subject, vibrating a 
steering wheel, seat belt or floor area associated with the vehide; activating an 
audible alarm and/or switching of bright lights In the cabin of the vehicle, and 

25 flashing hazard lights and/or sounding of an audible alarm such as siren or horn. 

30. A method according to claim 27 wherein the step of intervening Indudes 
interfacing to a remote monitoring station via a wireless communication system. 

30 31. Apparatus for detemnining vigilance state of a subject substantially as 
herein described with reference to any one of the embodiments Illustrated In the 
accompanying drawings. . 



wo 00/44580 PCT/AU99/01166 

-44- 

32. A method for determining a vigilance state of a subject substantially as 
herein described with reference to any one of the embodiments illustrated in the 
accompanying drawings. 
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